Ch5. Data Transformation(1)

5.2.4 Exercises

 

1. Find all flights that
   

1. Had an arrival delay of two or more hours

filter(flights, arr_delay >= 2)

b. Flew to Houston (IAH or HOU)

flights_departed <- filter(flights, !(is.na(dep_time))) # extract only the flights which left
filter(flights_departed, dest=="IAH" | dest=="HOU")

c. Were operated by United, American, or Delta

filter(flights_departed, carrier=="UA" | carrier=="AA" | carrier=="DL")

d. Departed in summer (July, August, and September)

filter(flights_departed, month == 7 | month == 8 | month == 9)

filter(flights_departed, month %in% c(7:9))

e. Arrived more than two hours late, but didn’t leave late

filter(flights_departed, arr_delay > 120 & !(dep_delay > 0))

f. Were delayed by at least an hour, but made up over 30 minutes in flight

filter(flights_departed, dep_delay >= 60 & (dep_delay - arr_delay) > 30)

g. Departed between midnight and 6am (inclusive)

filter(flights_departed, !(dep_time > 0600 & dep_time < 2400))

2. Another useful dplyr filtering helper is between(). What does it do? Can you use it to simplify the code needed to answer the previous challenges?

between() is a shortcut for x >= left & x <= right

Departed in summer (July, August, and September)

flights_departed[between(flights$month, 7, 9),]

Departed between midnight and 6am (inclusive)

flights_departed[!(between(flights_departed$dep_time, 601, 2359)),]

3. How many flights have a missing dep_time? What other variables are missing? What might these rows represent?

sum(is.na(flights$dep_time))

[1] 8255

head(filter(flights, (is.na(dep_time))))

Missing dep_time means that the flight didn’t depart. Therefore, the following variables do not exist in the flight.
Dep_time, dep_delay, arr_time, arr_delay and air_time

4. Why is NA ^ 0 not missing? Why is NA | TRUE not missing? Why is FALSE & NA not missing? Can you figure out the general rule? (NA * 0 is a tricky counterexample!)

NA ^ 0 # (anything) ^ 0 always returns 1

[1] 1

NA | TRUE  # '|' returns TRUE if either is TRUE and FALSE if both are FALSE.

[1] TRUE

NA | FALSE # returns the missing value because it is not the case above.

[1] NA

FALSE & NA # '&' returns TRUE if both are TRUE and FALSE if either is FALSE.

[1] FALSE

TRUE & NA # returns the missing value becasue it is not the case above .

[1] NA

NA * 0 # '*' requires two real numbers for the calculation

[1] NA

General rule) The calculation result including NA depends on each operator.

5.3.1 Exercises

1. How could you use arrange() to sort all missing values to the start? (Hint: use is.na()).

arrange(flights, is.na(dep_time)) # missing values to the end

arrange(flights, desc(is.na(dep_time))) # missing values to the start

2. Sort flights to find the most delayed flights. Find the flights that left earliest.

arrange(flights_departed, desc(dep_delay))

arrange(flights_departed, dep_delay)

the most delayed flight: HA 51 departed 1301 min later than scheduled
the flight that left earliest: B6 97 departed 43 min earlier than scheduled

3. Sort flights to find the fastest flights.

arrange(flights_departed, desc(distance/air_time))

DL 1499 travelled 762 miles in 65 min.

4. Which flights travelled the longest? Which travelled the shortest?

arrange(flights_departed, desc(distance))

arrange(flights_departed, distance)

The flights from JFK to HNL travelled 4983 miles.
The flights from EWR to LGA travelled 80 miles.

5.4.1 Exercises

1. Brainstorm as many ways as possible to select dep_time, dep_delay, arr_time, and arr_delay from flights.

select(flights, dep_time, dep_delay, arr_time, arr_delay)

select(flights, starts_with("dep"), starts_with("arr"))

2. What happens if you include the name of a variable multiple times in a select() call?

select(flights, dep_time, dep_time)

The variable value is returned only once.
3. What does the one_of() function do? Why might it be helpful in conjunction with this vector?

vars <- c("year", "month", "day", "dep_delay", "arr_delay")
select(flights, one_of(vars))

one_of() selects variables in character vector.

4. Does the result of running the following code surprise you? How do the select helpers deal with case by default? How can you change that default?

select(flights, contains("TIME")) # default: case insensitive

select(flights, contains("TIME", ignore.case = FALSE)) # case sensitive

default: ignore.case = TRUE

5.5.2 Exercises

1. Currently dep_time and sched_dep_time are convenient to look at, but hard to compute with because they’re not really continuous numbers. Convert them to a more convenient representation of number of minutes since midnight.

tempo <- mutate(flights,
  dep_minutes = dep_time %/% 100*60 + dep_time %% 100, 
  sched_dep_minutes = sched_dep_time %/% 100*60 + sched_dep_time %% 100) # convert into number of minutes  
select(tempo, dep_time, dep_minutes, sched_dep_time, sched_dep_minutes) # easy to calculate (in number of minutes)

2. Compare air_time with arr_time - dep_time. What do you expect to see? What do you see? What do you need to do to fix it?

a <- transmute(flights, air_time, difference = arr_time - dep_time)

Dep_time and arv_time depend on the timezone of departure and arrival airport. So we need to convert dep_time and arr_time into standardized time by applying local time zone and Daylight Saving Time(DST)
e.g. Davis UTC -7
New York UTC -4
London UTC -1

3. Compare dep_time, sched_dep_time, and dep_delay. How would you expect those three numbers to be related?

transmute(flights, dep_time, sched_dep_time, dep_delay)

dep_delay = dep_time - sched_dep_time

4. Find the 10 most delayed flights using a ranking function. How do you want to handle ties? Carefully read the documentation for min_rank().

arrange(flights, min_rank(desc(dep_delay)))

min_rank(): gives smallest values the small ranks, equivalent to rank(ties.method=“min”)

5. What does 1:3 + 1:10 return? Why?

1:3 + 1:10

longer object length is not a multiple of shorter object length

 [1]  2  4  6  5  7  9  8 10 12 11

The elements of the small vector are recycled until all elements of the long vector are manipulated by the operator.

1:3 + 1:9

[1]  2  4  6  5  7  9  8 10 12

6. What trigonometric functions does R provide?

Usage (x, y: numeric or complex vectors)

cos(x) cosine sin(x) sine tan(x) tangent

acos(x) arc-cosine asin(x) arc-sine atan(x) arc-tangent atan2(y, x) two-argument arc-tangent

cospi(x) cos(pix) sinpi(x) sin(pix) tanpi(x) tan(pi*x)